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° Troubleshooting 
° New features 
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Troubleshooting 


* Neighbor stability 

» Stuck-in-active routes 
» High CPU 

° Troubleshooting tools 
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Neighbor Process—Review 


° Multicast hellos 
224.0.0.10 


* Neighbor timers 
Hello Interval—5 or 60 sec. 
Hold time—15 or 180 sec. 
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Neighbor Process—Review 


RTRA#show ip eigrp neighbors 


IP-EIGRP neighbors for process 1 


www.cisco,.com 
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_ Common Neighbor Stability Problems _ 


° Physical link up/down 
° Hold timer expiration 
° Retry limit exceeded 


» Manual changes (sort of) 
° Stuck-in-active routes 


- Physical Link Up/Down ; 


° Interface reports the link down 
to EIGRP 


° EIGRP takes down the neighbors 
through this interface 


° Look in the logs for line up/down 
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Hold Timer Expiration 


* Hold time passed in Hello packet 


° If hold time expired, EIGRP packet 
not seen for hold time 


* Usually, means multicast Hellos are 
missed 


* Typically caused by congestion or 
physical errors 


Retry Limit Exceeded 


* Reliable packet was not acknowledged 


° Packet retransmitted based on RTO 
(6 X SRTT) 


Exponential backoff 
Max. RTO of 5000 ms, min of 200 ms 
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Retry Limit Exceeded (Cont.) 


° Actual timeout is 16 retransmits or 
hold time, whichever is longer 


16 retransmits takes between 50 seconds 
and 80 seconds 


For low-speed NBMA, holdtime is the 
deciding factor 


315 
0912_04F9_c3 © 1999, Cisco Systems, Inc. 


RTRA#show ip eigrp neighbors 

IP-EIGRP neighbors for process 1 

H Address Interface Hold Upti 
(sec 

2 10.1.1.1 Eto 
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| Manual Changes (Sort Of) 


* MTU changed 
* Summary changed 
* Route filter changed 
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cS Neighbor Stability Problems (Cont.) _ 


° Stuck-in-active routes 
Often very complex problems 
Will be covered in later section 
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Troubleshooting Tools for 
Neighbor Problems 


RouterA#config terminal 

Enter configuration commands, one per line. End with CNTL/Z. 
RouterA(config) #router eigrp 1 

RouterA(config-router) #eigrp log-neighbor-changes 
RouterA(config-router) #logging buffered 10000 
RouterA(config) #service timestamps log datetime msec 
RouterA(config) #4Z 

RouterA# 
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Log-Neighbor-Changes Messages 


Neighbor 10.1.1.1 (Ethernet0) is down: peer restarted 

Neighbor 10.1.1.1 (Ethernet0) is up: new adjacency 

Neighbor 10.1.1.1 (Ethernet0) is down: holding time expired 
Neighbor 10.1.1.1 (Ethernet0) is down: retry limit exceeded 
Neighbor 10.1.1.1 (Ethernet0) is down: route filter changed 
Neighbor 10.1.1.1 (Ethernet0) is down: interface delay changed 
Neighbor 10.1.1.1 (Ethernet0) is down: interface bandwidth changed 
Others, but not often... 
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Troubleshooting Tools for 
Neighbor Problems (Cont. 


rp-esc-2621b#debug eigrp packet hello 

EIGRP Packets debugging is on (HELLO) 

*Mar 16 19:08:38.521: EIGRP: Sending HELLO on Serial1/1 
*Mar 16 19:08:38.521: AS 1, Flags 0x0, Seq 0/0 idbQ 0/0 iidb¢ 
*Mar 16 19:08:38.869: EIGRP: Received HELLO 


*Mar 16 19:08:39.081: EIC 


*Mar 16 19:08:39.081: 
*Mar 16 19:08:39.749: E 
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Unusual Neighbor Problems 


Unidirectional links 
Mismatched masks 


Mismatch of primary/secondary 
addresses 
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Unidirectional Links 


Broken PVC 


RtrA#show ip eigrp neighbors 
IP-EIGRP neighbors for process 1 
rtrA# 


RtrB#show ip eigrp neighbors 

IP-EIGRP neighbors for process 1 

H Address _ Interface Hold Uptime SRTT RTO Q Seq 
(sec) (ms) Cnt Num 

110.1.102.2 Eto 14 00:00:15 0 5000 4 0O 
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interface serial 0 

ip address 10.1.1.2 255.255.255.128 
interface serial 1 

ip address 10.1.3.1 255.255.255.0 


Router 


interface serial 0 


ip address 10.1.1.1255.255.255.0 Interface ethernet 0 
interface ethernet 0 ip address 10.1.2.2 255.255.255.0 


ip address 10.1.2.1 255.255.255.0 interface serial 0 
ip address 10.1.3.2 255.255.255.0 


Router 


Routing Table: Routing Table: 
C 10.1.1.0/24 Serial 0 D 10.1.1.0/24 10.1.2.1 
D 10.1.1.0/25 10.1.2.2 D 10.1.1.0/25 10.1.3.1 
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~ Primary/Secondary Mismatch . 


Reports 50.1.1.3 as 
Neighbor with Retry 


; Limit Exceeded : 
Primary: 10.1.1.1/24 Primary: 10.1.1.2/24 
Secondary: 50.1.1.1/24 Secondary: 50.1.1.2/24 
Router Ss Routes nami 
Source = 10.1.1.1" Source = 10.1.1.2 


a 


Router = source = 50.1.1.3 
Cc encase 


No Sign of 10.1.1.1 or 7 Only Address - 50.1.1.3/24 
10.1.1.2 in Neighbor Table 


Not on Common Subnet! 
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Troubleshooting 


* Neighbor relationship 
° Stuck-in-active routes 
* High CPU 

* Troubleshooting tools 
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Stuck-In-Active Routes (SIA) . 


%DUAL- 3- SIA: Route 10. 64.5. o Z 25 


° aways indicates at least two problems 
* Two parts—Stuck and Active 
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Active Process 


» Normal process for resolving 
network changes 


* Normal (up) state is passive 


° Route “active” if lost and no other 
successor or feasible successor 
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Active Process (Cont.) 


Queries 


Replies 


rai” 


Active Process (Cont.) 


» Query process stops when: 
All queries are answered 
End of network reached 


End of the autonomous system 
reached (sort of) 


The lost component is unknown 
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Stuck-In-Active 


» When a route goes active, timer started 
Approximately 3 to 3-1/2 minutes 


° If timer expires without all queries 
being answered, “stuck” in the 
active process 


Stuck-in-Active (Cont.) 


° On the router where timer expires: 
Reinitializes neighbor(s) who didn’t answer 


Goes active on all routes known through 
bounced neighbor(s) 


Re-advertises to bounced neighbor all 
routes that we were advertising 
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_ Likely Causes for Stuck-In-Active _ 


° Bad or congested links 

* Query range is too long 

° Excessive redundancy 

* Router memory shortage 

° Software defects (very seldom) 


Troubleshooting SIA 


* Two parts—stuck and active 

* Need to troubleshoot both parts 
Cause of active often easier to find 
Cause of stuck more important to find 
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Troubleshooting 
wee tne Active PartofSiAs 


* Determine what is common to 
routes going active 


/32s from dial-in PPP? 
Flapping link(s)? 
From the same region of the network? 


Troubleshooting 
the Stuck Part of SIAs 


* Show ip eigrp topology active 
Useful only while the problem is 
occurring 


If problem isn’t occurring at the time, 
it is difficult to find the source of 
route getting stuck 
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Chasing Active Routes— 
Example 


Why Is RTRA Reporting SIA Routes? 
Let’s Look at a Problem in Progress... 


20.2.1.0/24 
iis 10.1.1.0/24 AIRE 


J 2 ; 10.1.2.0/24 


RTR 
2 id 


RTRA# show ip eigrp topology active | 
A 20.2.1.0/24, 1 successors, FD is Inaccessible 
1 replies, active 00:00:53, query-origin: Local origin 10.1.3.0/24 
via Connected (Infinity/Infinity), LoopbackO 
Remaining replies: 
via 10.1.1.2, r, Serial 
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_ Chasing Active Routes (Cont.) a: 


20.2.1.0/24 
RTRA 40.1.1.0/24 FIRB 
J 2 10.1.2.0/24 


RTR 
2 c 


RTRB#show ip eigrp topology active ees 


A 20.2.1.0/24, 1 successors, FD is Inaccessible A 
1 replies, active 00:01:04, query-origin: Successor Origin 
via 10.1.1.1 (Infinity/Infinity), Serial1 10.1.3.0/24 
Remaining replies: 
via 10.1.2.2, r, Serial 
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Chasing Active Routes (Cont.) 


20.2.1.0/24 
iia 10.1.1.0/24 RIES 


| 2 


RTRC#show ip eigrp topology active 
A 20.2.1.0/24, 1 successors, FD is Inaccessible 
1 replies, active 00:01:18, query-origin: Successor Origin 
via 10.1.2.1 (Infinity/Infinity), Serial1 
Remaining replies: 
via 10.1.3.2, r, Serial 
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20.2.1.0/24 
pice 10.1.1.0/24 ATE 


| 2 


RTRD#show ip eigrp topology active 
IP-EIGRP Topology Table for process 1 
RTRD# 
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10.1.2.0/24 


RTR 
2 id 


naa 


10.1.3.0/24 


10.1.2.0/24 


RTR 
2 c 


tae 


10.1.3.0/24 


~ Chasing Active Routes (Cont.) | 


20.2.1.0/24 
iia 10.1.1.0/24 RIES 


J 2 ; 10.1.2.0/24 


RTR 
2 id 


RTRC#show ip eigrp topology active ea 
A 20.2.1.0/24, 1 successors, FD is Inaccessible 1 
1 replies, active 00:01:48, query-origin: Successor Origin : 
via 10.1.2.1 (Infinity/Infinity), Serial1 
Remaining replies: 
via 10.1.3.2, r, Serial 


10.1.3.0/24 
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20.2.1.0/24 
pice 10.1.1.0/24 IRE 


J 2 1 10.1.2.0/24 
RTRC 


RTRC#show ip eigrp neighbor 

IP-EIGRP neighbors for process 1 

H Address Interface Hold Uptime SRTT RTO Q Seq 
(sec) (ms) Cnt Num 

1 10.1.3.2 Sei 10 00:00:13 0 5000 1 O 

0 10.1.2.1 Sed 12 5d20h 143 858 0 56 
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_ Chasing Active Routes (Cont) __ 


20.2.1.0/24 
iia 10.1.1.0/24 RIES 
1 2 


a 10.1.2.0/24 
RTRC 


RTRD#ping 10.1.3.1 
Type escape sequence to abort! 
Sending 5, 100-byte ICMP Echos to 10.1.3.1, timeout is 2 seconds: 


Success rate is 0 percent (0/5) 
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Troubles the Stuck Part 
__ Of SIAs (Cont.) | an 


° It’s not always this easy 
to find the cause 


*» Sometimes you chase the 
waiting neighbors in a circle 


If so, summarize and simplify! 
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Minimizing SIA Routes 


» Decrease query scope 
Summarization (manual or auto) 
Distribute-lists 

Particularly on dual-homed remotes 
Stub routers (future) 
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Decreasing Query Scope— 
_Example_ 


Distribution Layer Remote Sites 


; 10. 
Queries a 


Replies 


Router 
Cc 


Router 
D 


Router 
A 


Router 
E 
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Decreasing Query Scope— 
Summary 


» Remote routers are fully involved 
in convergence 


Usually remote routers are not 
intended to be transit paths 


Convergence is complicated through 
lack of information hiding 
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Decreasing Query Scope— 
__ ALittle Better _ 


Distribution Layer Remote Sites 


Queries 10.1% 47/24 
Replies C=) 
Router 
c 


IP summary-address eigrp 1 10.0.0.0 255.0.0.0 Router 
on All Outbound Interfaces toward Remote Routers E 
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Decreasing Query Scope— 
Summary 


» Convergence simplified 
Remotes immediately reply 
Compound by number of remotes! 


° Even simpler with new stub feature 


Decreasing Query Scope— 
2 Example _ 


Queries 


a= 


No Knowledge of —"—@ 1.8.0/24 


a 


IP summary-address eigrp 1 10.0.0.0 255.0.0.0 a 
on Outbound Interfaces toward Remote Routers 
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10.1.8.0/24 
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Minimizing SlA routes (Cont.) =—__ 


° Maintain reasonable redundancy 
Don’t make EIGRP’s job too difficult 
Use passive-interface 
Use hierarchy 
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c Removing Excessive Redundancy a 


Passive-Interface 
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: ___ Minimizing SIA Routes (Cont.) a 


° Multiple EIGRP AS’ are NOT 
the answer 


Terminates original query, 
but new one starts 


Adds redistribution complexity 


Requires distribute-lists to stop 
routing loops 


Multiple EIGRP AS’ a 


e= 


K 
(| EIGRP3 | 
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Impact of Low-Speed NBMA Links _ 


* Retry limit = hold time 

* Hold time = 180 seconds 

° Active timer = 180 seconds 

* One broken link can cause SIAs! 
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Low-Speed NBMA (Cont.) 


Queries 
Router 
Replies B 


>= 
Router 4 
A 
kes 
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__ Workarounds for Low-Speed NBMA _ 


° Use point-to-point subinterfaces 
instead 


* Change the Active timer to 4-5 
minutes (NOT recommended) 


* Change the hello/hold timers to 30/90 


* Neighbor relationship 


° Stuck-in-active routes 
° High CPU 


° Troubleshooting tools 
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Busy, Busy! Here’s a Big Chunk of It! 


RTRA#show process cpu 
CPU utilization for five seconds: 96%/35%; i e: 91%; five minutes: 89% 
PID Runtime (ms) Invoked i 5Min TTY Process 
60000 2484685 -00% Load Meter 
21595164 66420469 -17% IP-EIGRP Router 
30432980 414274 ~29% Check heaps 
1356 3043 00% Pool Manager 
(0) 2 00% Timers 
0 1 00% Handler 
0 1 - 00% Zone Manager 
0 al 00% Realm Manager 
2856 621291 00% 
2582920 6744442 00% 
te) 00% 


Seat Manager 
Input 
SERIAL A'detect 


FOU AID UAWNPR 
e970 00G0a0e ONO 
eo000000 0 

SCOoO0D OOOO O oO MO 
SCo0oCAGC0OGO0O Oe CO 


PR 
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High CPU (Cont.) 


Unstable Neighbors! 

RTRA#sh ip eigrp neighbor 
IP-EIGRP neighbors for process 1 
H Address Interface Hold Uptime SRTT RTO Q Seq 

(sec) (ms) Cnt Num 
11 10.1.15.181 Fd5/0 14 00:00:03 414 2484 35 309 
26 10.1.15.145 Fd5/0 14 00:01:10 102 612 1 495 
17. 10.1.15.130 Fd5/0 14 00:01:13 220 1320 2 241 

More unstable neighbors 

RTRA# 


Neighbor 10.1.1.1 (Ethernet0) is down: holding time expired 
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High CPU (Cont.) 


RTRA#show interface fddi 5/0 

Fddi5/0 is up, line protocol is up 
Hardware is cxBus FDDI, address is aa 
Description: Core FDDI Ring 
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High CPU (Cont.) 


° Show ip eigrp events 
Route(s) flapping? 
Routing loop? 


Find what activity is taking 
CPU and resolve it! 
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* Neighbor relationship 
» Stuck-in-active routes 
» High CPU 

¢ Troubleshooting tools 


~ EIGRP Troubleshooting Tools  _ 


* Debugs and the EIGRP event log 


On a busy, unstable network debugs 
can be hazardous to your health! 


Event log is non-disruptive— 
already running! 


Not for mere mortals to interpret! 
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Event Log 


* Always running (unless 
manually disabled) 


* Default 500 lines (configurable) 
* Most recent events at top of log 


° Three different event types 
can be logged 


EIGRP log-event-type 
[dual][xmit][transport] 


Default is dual—most useful 


Any combination of the three can be 
on at the same time 
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Event Log (Cont.) 


RTRA#show ip eigrp events 

Event information for AS 1: 

12:49:43.605 Poison squashed: 10.1.2.0/24 reverse 
12:49:43.553 Change queue emptied, entries: 1 
12:49:43.553 Metric set: 10.1.2.0/24 21536000 


ONODa kh WN = 
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SIA Event Log 


° Show ip eigrp sia 
Snapshot of event log when SIA occurs 


Typically lets you know results of SIA, 
not the cause 


Seldom useful 
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» Remember—can be dangerous! 
Use only in the lab or 
If advised by the TAC 
° To make a little safer: 
logging buffered <size> 
no logging console 


Debugs (Cont.) 


» Use modifiers to limit scope of 
route events or packet debugs 


debug ip eigrp AS network mask 
debug ip eigrp neighbor AS address 
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Debug IP Eigrp (Route Events) 


RTRA#debug ip eigrp 

IP-EIGRP Route Events debugging is on 

RTRA#debug ip eigrp neighbor 1 10.1.6.2 

IP Neighbor target enabled on AS 1 for 10.1.6.2 

IP-EIGRP Neighbor Target Events debugging is on 

RTRAj#clear ip eigrp neighbor 

RTRA# 

*Mar 17 15:50:53.244: IP-EIGRP: 10.1.6.0/24 - do advertise out Serial1/2 
*Mar 17 15:50:53.244: IP-EIGRP: Int 10.1.6.0/24 metric 20512000 -20000000 512000 
*Mar 17 15:50:53.244: IP-EIGRP: 10.1.8.0/24 - do advertise out Serial1/2 
*Mar 17 15:50:53.244: IP-EIGRP: Int 10.1.8.0/24 metric 28160 - 256002560 
*Mar 17 15:50:53.244: IP-EIGRP: 10.1.7.0/24 - do advertise out Serial1/2 
*Mar 17 15:50:53.244: IP-EIGRP: 10.1.1.0/24 - do advertise out Serial1/2 
*Mar 17 15:50:53.244: IP-EIGRP: Int 10.1.1.0/24 metric 28160 - 25600256 
*Mar 17 15:50:53.668: IP-EIGRP: Processing incoming UPDATE packet 
*Mar 17 15:50:54.544: IP-EIGRP: 10.1.6.0/24 - do advertise out Serial1/1 


315 
0912 04F9_c3 ® 1999, Cisco Systems, Inc. 


Debug IP Eigrp (Cont.) 


RTRA#debug ip eigrp 

IP-EIGRP Route Events debugging is on 

RTRA#debug ip eigrp 1 10.1.7.0 255.255.255.0 

IP Target enabled on AS 1 for 10.1.7.0/24 

IP-EIGRP AS Target Events debugging is on 

RTRAd#clear ip eigrp neighbor 

*Mar 17 15:52:20.940: IP-EIGRP: 10.1.7.0/24 - do advertise out Serial1/2 

*Mar 17 15:52:22.684: IP-EIGRP: 10.1.7.0/24 - do advertise out Serial1/1 

*Mar 17 15:52:22.684: IP-EIGRP: Int 10.1.7.0/24 metric 20512000 20000000 512000 
*Mar 17 15:52:22.940: IP-EIGRP: 10.1.7.0/24 - do advertise out Serial1/2 

*Mar 17 15:52:22.968: IP-EIGRP: Processing incoming UPDATE packet 

*Mar 17 15:52:24.684: IP-EIGRP: 10.1.7.0/24 - do advertise out Serial1/1 

*Mar 17 15:52:24.684: IP-EIGRP: Int 10.1.7.0/24 metric 20512000 - 20000000 512000 
*Mar 17 15:52:25.940: IP-EIGRP: 10.1.7.0/24 - do advertise out Serial1/2 
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Debug eigrp packet <type> 


RTRA#debug eigrp packet ? 
ack EIGRP ack packets 
hello EIGRP hello packets 
ipxsap EIGRP ipxsap packets 
probe EIGRP probe packets 
query EIGRP query packets 
reply EIGRP reply packets 
request EIGRP request packets 
retry EIGRP retransmissions 
terse Display all EIGRP packets except Hellos 
update EIGRP update packets 
verbose Display all EIGRP packet 
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Debug IP Eigrp Packet Terse 


RTRA#debug eigrp packet terse 
EIGRP Packets debugging is on 

(UPDATE, REQUEST, QUERY, REPLY, IPXSAP, PROBE, ACK) 
RTRA# 


*Mar 17 15:54:39.768: EIGRP: Enqueueing UPDATE on Serial1/2 nbr 10.1.7.2 iidbQ un/rely 
0/1 peerQ un/rely 0/0 serno 25-105 


*Mar 17 15:54:39.772: EIGRP: Requeued unicast on Serial1/2 

*Mar 17 15:54:39.772: EIGRP: Forcing multicast xmit on Serial1/2 

*Mar 17 15:54:39.772: EIGRP: Sending UPDATE on Serial1/2 nbr 10.1.7.2 

*Mar 17 15:54:39.772: AS 1, Flags 0x1, Seq 305/0 idbQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 
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Debug IP EIGRP Notificatio 
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Debug IP Eigrp Summary 


Summary 
Changed 


RTRA#debug ip eigrp summary 

IP-EIGRP Summary route processing debugging is on 
RTRA# 

*Mar 17 16:01:06.164: IP-EIGRP: conn_summary. 

*Mar 17 16:01:06.164: IP-EIGRP vary_de 

*Mar 17 16:01:06.168: IP-EIG| sum 

*Mar 17 16:01:06.168 EI 


This Is Why! 
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* Troubleshooting 
¢ New features 


315 
0912_04F9_c3 © 1999, Cisco Systems, Inc. 


Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 
0912 _04F9 c3.scr 


New Features 


° Classless network statements 
* Neighbor control 
° Stub routers 
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: Classless Network Statements 


router eigrp 1 
network 10.1.0.0 0.0 
network 192.3 


* Allows supernetted interfaces 


¢ Granular control of interfaces 
included in EIGRP 
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Neighbor Control 


router eigrp 1 
[no] eigrp neighbor auto-d 
[no] neighbor A.B.C. _— 


* Supports non-broadcast media 
(Classical IP on ATM) 


* Permits explicit definition 
of neighbors 
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Stub Routers 


Router eigrp 1 
stub [connected]|[static][summ 


_— 


¢ Defined on remote routers 


° Restricts route advertisement to 
connected, static, summary, or none 


° Queries are not propagated to 
stub routers 
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_ Decreasing Query Scope—Example 


Distribution Layer Remote Sites 
10.1.80/24 


Queries 
Replies > esta 


Router 


Router 
E 
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Decreasing Query Scope— 
Stub Remotes 


Distribution Layer Remote Sites 


Queries 10.1.80/24 
Replies 


Router 
D 


Router 
A 


Remote Routers (Router C, Router D, and Router E) Router 
Are All Defined as Stub Routers E 
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Advanced EIGRP Summary 


* There are many tools in EIGRP for 
troubleshooting network problems 


° New features will further improve 
EIGRP scalability and flexibility 
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Questions? 
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Please Complete Your 
Evaluation Form 


Session 315 
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